Trend log pusher

Introduction

Trend-log (TL) and trend-log-multiple (TLM) objects monitor a property / properties of a referenced object(s) and, when predefined conditions are met, saves ("logs") the value of the property and a timestamp in an internal buffer for subsequent retrieval. The data may be logged periodically, upon a change of value (COV) or when "triggered" by a write to the trigger property. The trigger property allows the acquisition of samples to be controlled by network write operations or internal processes. Errors that prevent the acquisition of the data, as well as changes in the status or operation of the logging process itself, are also recorded. Each timestamped buffer entry is called a trend log "record."

Trend log pusher is a part of Go-IoT's homogenous BACnet sensor to BACnet cloud solution.

The Trend log pusher is an APP intended for pushing trend log records from TL/TLM objects up to a cloud server via ftp or email (smtp). It supports pushing from local device TL/TLM objects as well as from any TL/TLM object on the BACnet network(s).
This APP can be configured to idiomatically sense TL and TLM objects in the BACnet network(s) and start collecting from those, or an administrator hand-pic TL/TLM objects to upload from.

The file format created and sent via ftp/smtp to a cloud server depends on file-format supported by the cloud server. Among those is a XML-format used by Go-IoT Cloud with full information such that the Go-IoT Cloud server can automaticly establish objects and time-series in it's database. This eliminates totally configuration work at server end.
Go-IoT strive to support new file format as requested by cloud-services. If a new format is needed please contact us at support@go-iot.io.

Configuring

For creating tha APP instance and configuring its general parameters refer to "APPs" in the DINGO-Manager documentation.

Parent ID: Usage of this integer identifier is local matter of the receiving cloud server.
Go-IoT Cloud uses it for looking up the unique ID of "gateway group" where under the data should be stored, when receiving the first file from the trend log pusher.

Site identifier: Usage of this text based identifier is local matter of the receiving cloud server. However this identifier is supposed to be unique among all sites sending in data files to the cloud server.

Site name: Usage is local matter of the receiving cloud server. It can be used for building name or what ever when the cloud server creates data-structure around the trend log pushing site.

File name initials: The first letters in the file-name uploaded by the trend log pusher. For further information about file name creation, refer to section "Supported file formats" below.

Server user ID: Specify cloud server user id, such that the receiving cloud server knows which user the file should be applied to.

File format: Specify one of the file formats in the drop-down list. If your cloud server does not support any of the file formats (see "Supported file formats" section below), contact Go-IoT Cloud at support@go-iot.io.

Markup language: From the drop-down list select one of the markup-language.

Note! The selected markup-language must be supported by the file format specified above.

Format: To control readability of the file or specify if cloud-server needs/supports it:
"Text extend enums and bit strings": If this check box is checked All BACnet enums and bit-strings are text extended. That is to say instead of writing the enum or bit-string as a number to file those are written as text. For example if checked, the object name of the from-logged as analog-input object is written as "analog-object" instead of the enum value "0". Refer to the ASHRE BACnet standard for further information about enums and bit-strings.
"Format with tabs and new lines": If checked tab- and new-line characters are added to the file to make it more human-readable.

Note! Check with your receiving cloud server if it can support the format-settings selected here.

Data polling frequency: The frequency in minutes the TL/TLM objects should be polled for new data.
Future option: The trend log pusher only polls the TL/TLM objects that have indicated new data ready event. This will eliminate polling of TL/TLM objects that have no new data.

Polling offset: Polling: Offset from aligned polling frequency in seconds. For example if Data polling frequency is set to 60 minutes and Polling offset to 300, data records will be polled from the TL/TLM objects 5 minutes after the whole hour.

Push retries: If pushing the data-file to the cloud server fails (lost connection, server down, etc.), this parameter specifies how many retries should be performed.

Push retry interval: For the push retries specified above this parameter specifies the interval between retries.

Push method(s): Currently only FTP is supported as protocol for pushing trend data files to a cloud server.

FTP server URL/IP: The URL or IP address of the FTP server to use.

FTP user ID: The user ID to login to the FTP server.

FTP password: The password to use for the FTP login.

FTP home directory: Home directory to upload to. Leave empty or as '/' if uploading to the root folder.

Currently no support for using emails.

Collect TL/TLM objects: Where and how to search for TL/TLM objects. There are 3 options in the drop-down box:
From network: Specify this if you want the trend log pusher to find all TL/TLM objects on the BACnet network(s) accessible in whatever BACnet device there might be out there.
From local device: If this is specified the trend log pusher will only search for TL/TLM object in the local device (the same device as the trend log pusher is running on).
From parameter list: From a hand picked list of TL/TLM objects specified in the Loggers property below.

Collect TL/TLM refresh interval: Specify in minutes the interval to search for new TL/TLM objects. Specify 0 to only collect new TL/TLMs at DINGO-Stack startup.

Loggers: Array of BACnetObjectReferences to TL/TLM objects that the trend log pusher is currently managing plus last timestamp of last log record for each of them pushed successfully to the cloud server.

Note! By rolling back this the timestamp you can enforce the trend-log-pusher to push again records that have already been pushed.
By advancing the timestamp however log records can be skiped for the TL/TLM object til the timestamp is reached.

Loggers black-list: Array of BACnetObjectReferences to TL/TLM object that should be black-listed, that is to say never included in the trend-log pushing.

If a TL/TLM object is black-listed that also exist in Loggers the object will not participate in the trend log pushing, as long as it exist in the black-list.

Supported file formats

An example file can be found here... briefly explained such:
<Composition name="bacnet-2-cloud-trend-log">...</Composition> refers to the "Log file properties" section in the paramters above.

<Composition name="app-device">...</Composition> contains information about the BACnet device where the trend log pusher is run.

<List name="devices">...</List> contains a list of trend log objects and their logged from devices.

<List name="tl-data">...</List> is the data section it self. Note that each trend log record list (<List name="values">) is identified with a uniqu identifier, like
<OctetString name="unique-idetnifier" value="0201A5E20000000200000055FFFFFFFF"/> which refers to the same identifier in "logged from devic" in the trend log list above.
The Identifier is built up as a hexacecimal sequence representing BACnetDeviceObjectPropertyreference such:
DeviceIdentifer , ObjectIdentifier, PropertyIdentifier, ArrayIndex
Each identifier in this sequence is zero padded 32 bit hexadecimal string. Remember that DeviceIdentifier and ObjectIdentifier are built up as ObjectType in the 10 most significant bits (10msb) and the ObjectInstance number in th 22 least significant bits (22lsb) according to the BACnet standard.
For the example above we can read out the individual identifiers as:

DeviceIdentifer = 0201A5E2h = ‭0000001000 0000011010010111100010‬b> ObjectType = 8 = device-object, ObjectInstance = 108002

ObjectIdentifier = 00000002h = 0000000000 0000000000000000000010b => ObjectType = 0 = analog-input, ObjectInstanse = 2

PropertyIdentifier = 00000055h = 85 = present-value

ArrayIndex = FFFFFFFFh = 'not an array'

To get a fully cloud unique identifier it is up to the cloud server receiver to prepend some information ahead of this id. Go-IoT Cloud does this by prepending the site-identifier found in the header of the file.